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METHOD FOR AD APT I VE L Y DERIVING EXPOSURE TIME AND 
FRAME RATE FROM IMAGE MOTION 

FIELD OF THE INVENTION 

The invention relates generally to a method for determining the 
exposure time and frame rate of an image capture system in order to minimize the 
motion blur between successive frames. 



BACKGROUND OF THE INVENTION 

10 FIG. 1 illustrates an image capture system, such as a camera, 

having an imaging lens 10 with a focal length f and an imaging sensor 12. An 
object 14 at a distance d Q in front of the camera will be properly focused at the 
imaging sensor 12 at a distance d[ behind the lens if the relationship between d 0 , 
dj and f is 
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— + — =!. (1) 
do dj f 



If an object 14 is moving at a constant velocity v (m/s) within the field of view 
(FOV) 16 of the scene, and the exposure time of the image capture system is At, 
20 then the object will appear smeared in the image over a distance s, given by 



s = — v At (meters) . (2) 
d„ 



If the imaging sensor 12 captures a digital image, then the object will be displaced 
25 by a distance s, measured in pixels, given by 



s = v At (pixels) , (3) 
d 0 P 
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where p is the distance between adjacent detector elements in the digital detector 
array. Also, if the image capture system moves with respect to the scene at a 
velocity v during the exposure time of the image, then the entire scene within the 
FOV 16 will be smeared over a distance s. 

The image smear can be reduced if the exposure time is decreased, 
but this will decrease the signal at the detector. If the signal is measured as the 
number of electrons produced by the object at the detector, then the signal is 
given by 



A ir At ^" ,nax 

= ^°p- ^ x > (4) 

4(f#) he , 



where X is the wavelength of light, is the transmittance of the optical system, 
L object is the spectral radiance of the object, r\ is the detector quantum efficiency, 
1 5 A detectQr is the area of the detector element, f# is the optical system f-number, h is 
Planck's constant, and c is the speed of light. Random noise arises from elements 
that add uncertainty to the signal level of the target and is quantified by the 
standard deviation of its statistical distribution a noise . The signal-to-noise ratio 
(SNR) of the image is given by 



20 



SNR _ mean object signal _ s object ^ 
signal deviation c noise 



Reducing the exposure time will decrease the image smear, but will also reduce 
the signal and the SNR of the image, generally resulting in an image that appears 
25 noisier. 

Referring to FIG. 2, when a fast moving object 14 at a velocity v is 
imaged with a video capture system, the object 14 will be smeared by a distance s 
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and the object 14 will be displaced between the two successive images, i.e. the 
first image frame 18 and the second image frame 20, by a distance Ad, given by 

Ad = -^-vt clock (pixels), (6) 
d oP 

5 

where t,.,,^ is the time interval between each successive image. Here a video 
capture system will be defined as an image capture system that acquires a 
sequence of images separated in time determined by the frame rate (FR). The 
frame rate is related to t CIOCk by 

10 

FR = — — (frames/s), (7) 

^ clock 

andt clock > At. 

If the object displacement and the smear are large enough, then the 
1 5 utility of the video images will be reduced. Longer exposure times are desired to 
increase the SNR but may cause an unacceptable smear of the object. If the smear 
is large enough, identification of the object may not be possible. Motion smear 
above two pixels will cause a loss of information that is generally not recoverable. 
Lower frame rates are desired to reduce the amount of image data that must be 
20 captured, disseminated, and stored, but may not capture enough frames of a fast 
moving object before it passes through the camera's FOV. Traditional electronic 
video image capturing devices have a predefined exposure time, or shutter time, 
and readout each image at a predefined frame rate. 

One technique used to acquire high-speed photographs that reduces 
25 the image motion smear without producing noisy images is to use a very fast 

illumination flash, such as a strobe, to decrease the effective exposure time. One 
such method proposed in U.S. Patent 4,918,522 uses a short light impulse to . 
image a quick-moving object by means of a video method. This is ideal in that it 
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produces a high signal over a very short exposure time. If a strobe is used, then it 
must be synchronized with the frame rate of the video capture system. 
Unfortunately, it is not always convenient or even possible to have a high 
intensity flash source available while acquiring images. For example, image 
5 capture systems used for remote sensing can not use a flash source. It is desirable 
then to capture an image with the longest exposure time possible, to maximize the 
SNR, but that does not produce an undesirable amount of image motion smear. 

With video capture systems, short exposure times can be used to 
minimize the image motion blur, and then successive frames can be added 

10 together to increase the signal. Fast moving objects in the scene can still produce 
artifacts in the reproduced image. The method proposed in U.S. Patent 4,652,907 
adaptively controls the parameters used in a recursive filter to sum the successive 
frames together to minimize the artifacts, but does not adaptively control the 
exposure time. Another method proposed in U.S. Patent 4,882,629 adaptively 

1 5 controls the exposure time based on the signal level, but does not account for the 
motion of objects in the scene. 

There is a need therefore for an image capture system that 
adaptively determines the best exposure time and frame rate based on the detected 
motion during the imaging sequence, thus keeping the image motion smear and 

20 the object displacement at or below a predetermined level. 

SUMMARY OF THE INVENTION 

The present invention is directed to overcoming one or more of the 
problems set forth above. Briefly summarized, according to one aspect of the 

25 present invention, the invention resides in a method for adaptively determining 

the exposure time for each frame captured by an image capture system of the type 
having an image sensor, optics for forming an image of a scene on the image 
sensor, and an image processor for determining an exposure time for the image 
sensor. More specifically, one aspect of the invention includes the steps of: a) 

30 acquiring two time-separated image frames during an initial exposure time set for 
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each of the image frames, wherein each image frame includes one or more 
moving objects from the scene; b) spatially registering tfie two image frames; c) 
measuring spatial displacements between the two image frames for the same 
moving objects; d) determining a fastest moving object in the scene from the 
5 measured spatial displacements; e) calculating a speed of the fastest moving 
object; f) calculating object smear for the fastest moving object based on the 
initial exposure time; g) if the object smear exceeds a maximum desired smear, 
calculating a new exposure time; and h) setting the exposure time for a subsequent 
image frame to the new exposure time. 

10 In yet another aspect of the invention, the method adaptively 

determines the frame rate of a system of the type described above by acquiring the 
two image frames according to an initial frame rate, then performing steps b) 
through e) as described above and further calculating the object displacement at 
the initial frame rate; then, if the object displacement exceeds a maximum desired 

1 5 displacement, a new frame rate is calculated and the frame rate for a subsequent 
image frame is set to the new frame rate. In yet a further aspect of the invention, 
the two methods may be combined to adaptively determine both the exposure 
time and frame rate of the image capture system. 

20 ADVANTAGEOUS EFFECT OF THE INVENTION 

This invention allows the exposure time and frame rate for each 
frame of a video image capture system to be automatically determined. The 
exposure time is determined such that the signal-to-noise ratio (SNR) is 
maximized while keeping the motion blur from objects moving in the scene to a 
25 predetermined amount. The frame rate is determined such that the object 
displacement does not exceed a predetermined amount. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a schematic of an image capture system useful in 
30 describing the background of the present invention. 
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FIG. 2 illustrates the smear and displacement of a moving object 
between two image frames. 

FIG. 3 is a schematic diagram of an adaptive video exposure time 
and frame rate system according to the present invention. 
5 FIG. 4 shows the image processing steps for determining the best 

exposure time and frame rate. 

FIG. 5 shows a schematic of the sensor processor. 

DETAILED DESCRIPTION OF THE INVENTION 

10 Because image processing systems are well known, the present 

description will be directed in particular to attributes forming part of, or 
cooperating more directly with, the method in accordance with the present 
invention. Attributes not specifically shown or described herein may be selected 
from those known in the art. In the following description, a preferred 

1 5 embodiment of the present invention would ordinarily be implemented as a 

software program, although those skilled in the art will readily recognize that the 
equivalent of such software may also be constructed in hardware. Given the 
method as described according to the invention in the following materials, 
software not specifically shown, suggested or described herein that is useful for 

20 implementation of the invention is conventional and within the ordinary skill in 
such arts. 

If the invention is implemented as a computer program, the 
program may be stored in conventional computer readable storage medium, which 
may comprise, for example; magnetic storage media such as a magnetic disk 
25 (such as a floppy disk or a hard drive) or magnetic tape; optical storage media 
such as an optical disc, optical tape, or machine readable bar code; solid state 
electronic storage devices such as random access memory (RAM), or read only 
memory (ROM); or any other physical device or medium employed to store a 
computer program. 
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Referring to FIG. 3, an image capture system according to the 
present invention includes imaging optics 10, a digital image sensor 12, an image 
processor 22, and a sensor processor 24. The image processor 22 in FIG. 3 



processes the images to determine the best exposure time and frame rate. The 
5 flow diagram for the image processor 22 is shown in FIG. 4. Let Ato and FRq be 
the initial exposure time and frame rate desired if no image motion information is 
available. Generally Ato * s 311 exposure time long enough to provide an image with 
a sufficient SNR, and FR<, is a frame rate low enough to minimize the image data 
that needs to be captured. 

10 The method for determining the best frame rate and exposure time 

according to the present invention first acquires two successive frames from the 
image capture system, the first frame 1 8 denoted by Ii(x,y) and the second frame 
20 denoted by I 2 (x,y), where x and y are the spatial coordinates in pixels. The 
images are separated by a time interval t' clock and a frame rate FR -l/f clock . The 

1 5 . exposure time and frame rate used for the very first two frames of a video 
sequence is A to and FRq. 



so Ii(x,y) and I 2 (x,y) must be spatially registered 26 to remove the displacement 
that may have occurred between the frames. The method in the preferred 
20 embodiment for spatially registering the two scenes calculates a correlation matrix 



where N x is the number of pixels in the x-direction and N y is the number of pixels 
25 in the y-direction of I^y) and I 2 (x,y). The values for Ax and Ay at which C(Ax, 
Ay) is a maximum, Ax,^ and Ay^, are the values of the scene displacement in 
the x-direction and the y-direction respectively. A spatially registered image 
r 2 (x,y) is calculated by 



The camera pointing may have changed between the two frames, 



C(Ax, Ay) = £ £ I, (x +Ax, y + Ay)I 2 (x, y) , 



(8) 



*=1 y=l 
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T 2 (x, y) = I 2 (x - A Xmax , y - Ay „) . (9) 

This method will only remove translation differences between the two frames. 
More robust frame registration algorithms can be used, e.g. affine 
5 transformations, especially if the image scale and rotation changes between the 
two frames (See Digital Video Processing by Murat Tekalp, Prentice Hall, 1995). 
Imaging acquisition parameters can also be used to predict the camera motion, 
scale and rotation changes. 

Next, a means for detecting object motion between I^x^y) and 

10 r 2 (x,y) is employed 28. In the preferred embodiment a sub-block correlation 

algorithm is used. The image Ii(x,y) is subdivided into sub-blocks, each of size N s 
x N s pixels. The sub-blocks need to be small enough to allow the motion of 
individual objects to be detected, but large enough to have statistically significant 
calculations. In the preferred embodiment, N s =16. Each sub-block Is(x,y) is 

15 correlated with F 2 (x,y), 



C B ( Ax, Ay) = £ f; I. (x +Ax, y + Ay)I' 2 (x, y) (10) 

x==l y=l 



The values for Ax and Ay at which C s (Ax, Ay) is a maximum are the values of the 
20 object displacement in the x-direction and the y-direction respectively for that 
sub-block. The distance, in pixels, that an object moved between Ii(x,y) and 
P 2 (x,y) is given by 



Ad = ^Ax 2 +Ay 2 . (11) 

25 

The displacement of each moving object detected is calculated using the other 
sub-blocks and the largest displacement of an object between the two frames is set 
equal to Ad^^ 30. The speed of the fastest moving object, in pixels per second, is 
calculated 32 by 



9 



v max = Ad rastest FR 1 . (12) 

If the frame rate FRo is used for the next frame I 3 (x,y), then the amount the 
object will be displaced between the first image frame 18 and the second image 
frame 20 can be calculated 34 by 



Ad 0 = Vmax -^. (13) 



10 The displacement Ado is compared 36 to the maximum displacement desired, Ad^. If 
Ac^ ^ Admax> then the frame rate FRo is used 38 for the next frame I 3 (x,y) of the video 
collection. In the preferred embodiment, Ad^ is less than ten pixels, e.g. eight pixels. If 
Ado > Admax> then a new frame rate is calculated 40 by 



15 FR = Ymax , (14) 

Ad_ 



and the frame rate FR is used 42 for the next frame I 3 (x,y) of the video collection. 

If the exposure time Ato is used for the next frame I 3 (x,y), then the 
amount of motion smear from the moving object can be calculated 44 by 



20 



At 0 (pixels). (15) 



The smear s 0 is compared 46 with the maximum smear desired, s max . If s 0 < s max , where 
s^ is the maximum smear desired, then the exposure time Ato is used 48 for the next 
25 frame I 3 (x,y) of the video collection. In the preferred embodiment, s max is less than two 
pixels, e.g. 1.5 pixels. If s 0 > s^, then a new exposure time is calculated 50 by 
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At = £max_ 5 (16) 



and exposure time At is used 52 for the next frame I 3 (x,y) of the video collection. 

The sensor processor 24 in FIG. 3 is used to capture the image data 
5 from the digital image sensor 12 at an exposure time and frame rate that has been 
determined by the image processor 22. A schematic of the sensor processor 24 is 
shown in FIG. 5. The signal from the frame rate clock 54 determines the intervals 
at which the voltage levels in the digital image sensor 12 change, thus 
determining the frame rate. For an electronic shutter the signal from the exposure 
10 time clock 56 will vary the voltage levels such that the detector charge is collected 
for the desired time At. For faster performance the sensor processor can be 
designed such that only N possible exposure times are allowed, i.e. 



At n =^t clock , (17) 

15 

where n and N are integers, n < N, and the desired At is rounded to the nearest At^ 
value. The same may be true of the frame rate, i.e., the sensor processor can be 
designed such that only N possible frame rates are allowed. In practice, the new 
frame rate and exposure time may be selected from a predetermined set of frame 

20 rates and exposure times stored in a database, based on rounding the desired 
values to the nearest values in the database. Although an electronic shutter is 
described in the preferred embodiment, a mechanical shutter can be used to 
provide the desired exposure time and frame rate. 

After the next frame is acquired with the desired exposure time and 

25 frame rate, the procedure outlined above is repeated for the next image frame, 

where I 2 (x,y) above becomes Ij(x,y), and the frame just captured becomes I 2 (x,y). 

In summary, in a method for determining the frame rate and 
exposure time for each frame of a video collection, the present invention requires 
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the image capture system to acquire at least two successive frames of a scene, 
separated in time. The two images are compared to determine if objects in the 
scene are in motion. If motion is detected, then the speed and displacement of the 
objects that are moving is determined. If the speed of the fastest moving object 
5 creates an unacceptable amount of image displacement, then the frame rate for the 
next frame is changed to one that produces an acceptable amount of image 
displacement. Also, if the speed of the fastest moving object creates an 
unacceptable amount of motion blur, then the exposure time for the next frame is 
changed to one that produces an acceptable amount of motion blur. The exposure 
10 time is changed using a mechanical or electronic shutter. 

The invention has been described with reference to a preferred 
embodiment. However, it will be appreciated that variations and modifications 
can be effected by a person of ordinary skill in the art without departing from the 
scope of the invention. 
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first image frame 
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image processor 
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register frames step 
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detect moving object step 
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determine Ad faslt!St step 
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calculate v^ step 
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calculate object displacement Ado step 
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compare Ado ste P to Ad,^ step 
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set frame rate to FRo step 
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calculate new frame rate step 
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set frame rate to FR step 
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calculate object smear s 0 step 
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compare s 0 step to s^ step 
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set exposure time to Ato step 
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calculate new exposure time step 
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set exposure time to At step 
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frame rate clock 
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exposure time clock 



